Compact Eye-Tracking Camera Systems and Methods

ABSTRACT

Systems and methods are provided for a compact eye-tracking camera assembly. The camera incorporates a lens and image sensor whose principal axis is parallel to the plane of the display surface, and a compound angle mirror configured to redirect incident IR light along the principal axis. The compound angle mirror has a first angle of approximately 45 degrees relative to a first axis, and a second angle of approximately 15-22 degrees (e.g., 18 degrees) relative to a second axis that is orthogonal to the first axis. The first and second axes are substantially parallel to the plane of the display surface.

TECHNICAL FIELD

The present invention relates, generally, to eye-tracking systems andmethods and, more particularly, to compact camera assemblies used inconnection with such eye-tracking systems.

BACKGROUND

Eye-tracking systems, such as those used in conjunction with desktopcomputers, laptops, tablets, head-mounted displays and other suchcomputing devices that include a display, generally incorporate one ormore illuminators for directing infrared light to the user's eyes, and acamera assembly (including a lens and image sensor) for capturingreflected images of the user's face for further processing. Bydetermining the relative locations of the user's pupils and the cornealreflections in the reflected images, the eye-tracking system canaccurately predict the user's gaze point on the display.

Recent advances in display technology as well as consumer demand forlightweight devices have driven a trend toward very thin displays,particularly in laptop computers and tablets. Indeed, some modern laptopdisplays have a thickness of 15 mm or less. This presents a challengewhen attempting to incorporate eye-tracking functionality in suchdevices, as the lenses and cameras used for eye-tracking generally havean irreducible minimum focal length, sometimes on the order of 15 mm ormore. While it is possible for lens/camera assemblies to be configuredsuch that they extend out of the front surface bezel, such non-flushconfigurations are generally undesirable for a variety of design andmanufacturability reasons. Systems and methods are therefore needed thatovercome these and other limitations of the prior art.

SUMMARY OF THE INVENTION

Various embodiments of the present invention relate to systems andmethods for, inter alia: i) a compact, laterally-mounted lens and imagesensor that receive incident IR light via a compound mirror generallyoriented toward a user's face while interacting with a display; i) acompact eye-tracking camera assembly incorporating a lens and imagesensor whose principal axis is parallel to the plane of the displaysurface, and a compound angle mirror configured to redirect incident IRlight along the principal axis; ii) the use of a compound angle mirrorthat has a first angle of approximately 45 degrees relative to a firstaxis, and a second angle of approximately 15-22 degrees (e.g., 18degrees) relative to a second axis that is orthogonal to the first axis,wherein the first and second axes are substantially parallel to theplane of the display surface.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The present invention will hereinafter be described in conjunction withthe appended drawing figures, wherein like numerals denote likeelements, and:

FIG. 1 is a conceptual overview of a computing device and eye-trackingsystem in accordance with various embodiments;

FIGS. 2A and 2B are front and side views, respectively, of a userinteracting with an eye-tracking system in accordance with variousembodiments;

FIG. 2C illustrates the determination of pupil centers (PCs) and cornealreflections (CRs) in accordance with various embodiments;

FIGS. 3A, 3B, and 3C are schematic diagrams of camera assembly geometryin accordance with various embodiments;

FIG. 4A is an isometric overview of a camera assembly in accordance withone embodiment;

FIG. 4B is an alternate, partial cut-away view of the camera assembly ofFIG. 4A;

FIG. 5 is an isometric overview of an eye-tracking assembly inaccordance with one embodiment;

FIGS. 6 and 7 illustrate the incorporation of the eye-tracking assemblyof FIG. 5 into a finished computing device in accordance with oneembodiment; and

FIG. 8 illustrates a compact lens enclosure structure in accordance withone embodiment.

DETAILED DESCRIPTION OF PREFERRED EXEMPLARY EMBODIMENTS

The present subject matter relates to systems and methods for a compact,laterally-mounted lens and image sensor that receive IR light via acompound mirror. Since the effective width (e.g., diameter) of suchcomponents are generally less than the overall length necessitated bythe lens focal length, by mounting the lens and image sensor parallel tothe display (rather than orthogonal) and using compound mirror tocollect the incident IR light, the camera assembly can be incorporatedinto much thinner displays.

As a preliminary matter, it will be understood that the followingdetailed description is merely exemplary in nature and is not intendedto limit the inventions or the application and uses of the inventionsdescribed herein. Furthermore, there is no intention to be bound by anytheory presented in the preceding background or the following detaileddescription. In the interest of brevity, conventional techniques andcomponents related to eye-tracking algorithms, image sensors, lensdesign, and digital image processing may not be described in detailherein.

Referring first to FIG. 1 in conjunction with FIGS. 2A-2C, the presentinvention may be implemented in the context of a system 100 thatincludes a computing device 110 (e.g., a desktop computer, tabletcomputer, laptop, smart-phone, head-mounted display, television panels,dashboard-mounted automotive systems, or the like) having a display 112and an eye-tracking assembly 120 coupled to, integrated into, orotherwise associated with device 110. It will be appreciated thatembodiments of the present invention are not limited to the particularshape, size, and type of computing devices 110 illustrated in thefigures.

Eye-tracking assembly 120 includes one or more infrared (IR) lightsources, such as light emitting diodes (LEDs) 121 and 122 (alternativelyreferred to as “L1” and “L2” respectively) that are configured toilluminate the facial region 281 of a user 200, while one or more cameraassemblies (e.g., camera assembly 125) are provided for acquiring, at asuitable frame-rate, reflected IR light from user's facial region 281within a field-of-view 270. Eye-tracking assembly may include one ormore processors (e.g., processor 128) configured to direct the operationof LEDs 121, 122 and camera assembly 125. Eye-tracking assembly 120 ispreferably positioned adjacent to the lower edge of screen 112 (relativeto the orientation of device 110 as used during normal operation).

System 100, utilizing computing device 110 (and/or a remote cloud-basedimage processing system) then determines the pupil centers (PCs) andcorneal reflections (CRs) for each eye—e.g., PC 211 and CRs 215, 216 forthe user's right eye 210, and PC 221 and CRs 225, 226 for the user'sleft eye 220. The system 100 then processes the PC and CR data (the“image data”), as well as available information regarding the headposition/orientation for user 200, and determines the location of theuser's gaze point 113 on display 112. The gaze point 113 may becharacterized, for example, by a tuple (x, y) specifying linearcoordinates (in pixels, centimeters, or other suitable unit) relative toan arbitrary reference point on display screen 112. The determination ofgaze point 113 may be accomplished in a variety of ways, e.g., throughthe use of eye-in-head rotations and head-in-world coordinates togeometrically derive a gaze vector and its intersection with display112, as is known in the art.

In general, the phrase “eye-tracking data” as used herein refers to anydata or information directly or indirectly derived from an eye-trackingsession using system 100. Such data includes, for example, the stream ofimages produced from the users' facial region 281 during an eye-trackingsession (“image data”), as well as any numeric and/or categorical dataderived from the image data, such as gaze point coordinates, cornealreflection and pupil center data, saccade (and micro-saccade)information, and non-image frame data. More generally, such data mightbe include information regarding fixations (phases when the eyes arestationary between movements), saccades (rapid and involuntary eyemovements that occur between fixations) scan-path (series of shortfixations and saccades alternating before the eyes reach a targetlocation on the screen), duration (sum of all fixations made in an areaof interest), blink (quick, temporary closing of eyelids), and pupilsize (which might correlate to cognitive workload, etc.).

In some embodiments, image data may be processed locally (i.e., withincomputing device 110 and/or processor 128) using an installed softwareclient. In some embodiments, however, eye tracking is accomplished usingan image processing module remote from computing device 110—e.g., hostedwithin a cloud computing system communicatively coupled to computingdevice 110 over a network (not shown). In such embodiments, the remoteimage processing module performs all or a portion of the computationallycomplex operations necessary to determine the gaze point 113, and theresulting information is transmitted back over the network to computingdevice 110. An example cloud-based eye-tracking system that may beemployed in the context of the present invention is illustrated in U.S.patent application Ser. No. 16/434,830, entitled “Devices and Methodsfor Reducing Computational and Transmission Latencies in Cloud Based EyeTracking Systems,” filed Jun. 7, 2019, the contents of which are herebyincorporated by reference.

Referring now to the conceptual block diagrams of FIGS. 3A and 3B, thegeometry of an exemplary camera assembly 125 will now be described. Moreparticularly, FIG. 3A is a side view of assembly 125, and FIG. 3B is ahead-on view of assembly 125. The coordinate axes illustrated in FIGS.3A and 3B correspond to those in FIGS. 1 and 2B. The z-axis generallyextends outward from the plane defined by display 112, and may thereforebe referred to as the normal vector of display 112. The x and y-axesboth lie in a plane parallel to display 112, wherein the x-axis isparallel to the major axis of eye-tracking assembly 120 (defined by aline extending through IR LEDs 121 and 122), and the y-axis isperpendicular to the x-axis.

With continued reference to FIG. 3A, incident IR light 340 (i.e.,reflected from the user's face) is reflected by a mirror 331 (e.g., anIR mirror or “hot mirror”) onto an image sensor (e.g., a CMOS sensor)334 via a lens 332 with a suitable focal length. Lens 332 may besupported by a lens barrel (or other such housing) 333. The centers ofimage sensor 334, lens barrel 333, lens 332, and mirror 331 aregenerally colinear along a principal axis 360, as shown. Thus, assembly125 is laterally or “sideways-mounted” in the sense that the principalaxis 360 lies within a plane parallel to the plane of display 112,rather than orthogonal (parallel to the surface normal) as wouldtraditionally be the case. In the illustrated embodiment, the principalaxis 360 is parallel to the x-axis (i.e., parallel to the lower edge ofdisplay 112, as depicted in FIG. 1).

As shown in FIG. 3A, mirror 331 is oriented at an angle 351 relative tothe x-axis—i.e., rotated about the y-axis by an angle θ—wherein θ issubstantially equal to 45 degrees (e.g., 45°±2°). At the same time, asshown in FIG. 3B, mirror 331 is oriented at an angle 352 relative to thez-axis by an angle Φ—wherein Φ is in the range of 15-20 degrees (e.g.,18°±1°).

Stated more intuitively, mirror 331 is positioned at a compound anglethat results in camera assembly 125 facing the user at a slightly upwardangle Φ. This is illustrated, for example, in FIG. 2B. Thus, the45-degree angle compensates for the camera assembly being mountedlaterally (along the x-axis), and the 15-20 degree angle accounts forthe usual position and orientation of a user's face during normaloperation of device 110.

In one embodiment, the diameter of lens 332 and/or lens barrel 333ranges from 8 to 12 mm (e.g., 10 mm), and required total track length(i.e., distance along the x-axis from the rightmost surface of lens 332to the surface of image sensor 334 ranges from 12 to 20 mm (e.g., 15mm). In such an embodiment, the resulting thickness required for device110 to accommodate camera assembly 125 is reduced by up to 50%. In oneembodiment, as shown in FIG. 3C, the image sensor 334 is also rotatedabout the x-axis (and mounted to a PCB 339) to compensate for therotation of mirror 331.

FIG. 4A is an isometric overview of a camera assembly in accordance withone embodiment, and FIG. 4B is an alternate, partial cut-away view ofthe same embodiment. Camera assembly 400 includes a housing 420, acompound angle mirror 401, a lens 402, a lens barrel 403, and an imagesensor 404—all of which are substantially colinear along a principalaxis 460. Assembly 400 might also include other sensors, such as adistance sensor 405 facing outward (i.e., the positive z axis). Asdepicted in FIG. 4B, mirror 401 has an orientation characterized byorthogonal angles 1 and 0. The track length d extends along the x-axisfrom the leftmost edge of lens 402 to the surface of image sensor 404.

FIG. 5 is an isometric overview of an assembly 500 that includes acamera assembly as depicted in FIG. 4. That is, camera assembly 400 ispositioned at the midpoint of a substrate (e.g., a plastic strip) 501.Antipodal printed circuit boards (PCBs) 511 and 512 are secured to theendpoints of substrate 501 (at a slight inward-facing and upward-facingangle, as shown) and are configured to mechanically and electricallyinterface with a pair of IR LEDs (e.g., IR LEDs 121 and 122 in FIG. 1).A circuit board 502 may also be provided to support the variouscontrollers, image processor chips, and other electronics used tocontrol the components of camera assembly 400 and PCBs 511, 512.

FIGS. 6 and 7 illustrate the incorporation of the eye-tracking assemblyof FIG. 5 into a finished computing device 700 in accordance with oneembodiment. As shown, assembly 500 is positioned between a backenclosure 601 and a display screen component 602 that are configured tobe secured together such that module the camera module 400 is framed byan opening 620 in component 602 (allowing a clear view of the user). Arear portion of module 400 may also be configured to fit within anopening 610 in back enclosure 601 such that module 400 is substantiallyflush therewith. While not illustrated in FIGS. 6 and 7, a platecomponent that is substantially transparent to IR light and at leastpartially opaque to visible light is preferably placed over or insertedwithin module 400 to protect its internal components while stillproviding the required functionality.

FIG. 8 illustrates a compact lens enclosure 806 in accordance with oneembodiment. In particular, as one of the goals of the present inventionis to provide a lens/mirror assembly that can be deployed in the contextof a thin display, the embodiment shown in FIG. 8 is advantageous inthat the enclosure 806 for lens 804 is, in at least one dimension, nogreater than the diameter of lens 804. Thus, a point 811 on the outerperimeter of lens 804 is substantially flush with the top surface 810 ofenclosure 806.

In summary, what has been described are various systems and methods fora compact eye-tracking camera assembly. In accordance with oneembodiment, a compact eye-tracking camera assembly is provided for acomputing device having a display. The eye-tracking camera assemblyincludes an infrared image sensor; a lens component; and mirrorcomponent. The infrared image sensor, lens component, and mirrorcomponent are substantially colinear along a principal axis; and themirror component is oriented at a compound angle configured to redirectincident IR light along the principal axis, the compound anglecharacterized by a first angle of approximately 45 degrees relative to afirst axis, and a second angle of approximately 15 to 22 degreesrelative to a second axis that is orthogonal to the first axis, thefirst and second axes being substantially parallel to a plane defined bythe display.

A tablet computing device in accordance with one embodiment includes anenclosure; a display coupled to enclosure; and an eye-tracking cameraassembly incorporated into the display, the eye-tracking camera assemblyincluding: an infrared image sensor; a lens component; and a mirrorcomponent. The infrared image sensor, lens component, and mirrorcomponent are substantially colinear along a principal axis, and themirror component is oriented at a compound angle configured to redirectincident IR light along the principal axis, wherein the compound angleis characterized by a first angle of approximately 45 degrees relativeto a first axis.

A method of manufacturing a tablet computing device in accordance withone embodiment generally includes providing an enclosure; providing adisplay; and forming an eye-tracking camera assembly including aninfrared image sensor, a lens component, and a mirror component, all ofwhich are substantially colinear along a principal axis. The methodfurther includes interconnecting the enclosure, display, andeye-tracking camera assembly such that such that the mirror component isoriented at a compound angle configured to redirect incident IR lightalong the principal axis, the compound angle characterized by a firstangle of approximately 45 degrees relative to a first axis substantiallyparallel to a plane defined by the display.

Embodiments of the present disclosure may be described in terms offunctional and/or logical block components and various processing steps.It should be appreciated that such block components may be realized byany number of hardware, software, and/or firmware components configuredto perform the specified functions. For example, an embodiment of thepresent disclosure may employ various integrated circuit components,e.g., memory elements, digital signal processing elements, logicelements, look-up tables, or the like, which may carry out a variety offunctions under the control of one or more microprocessors or othercontrol devices.

In addition, the various functional modules described herein may beimplemented entirely or in part using a machine learning or predictiveanalytics model. In this regard, the phrase “machine learning” model isused without loss of generality to refer to any result of an analysisthat is designed to make some form of prediction, such as predicting thestate of a response variable, clustering patients, determiningassociation rules, and performing anomaly detection. Thus, for example,the term “machine learning” refers to models that undergo supervised,unsupervised, semi-supervised, and/or reinforcement learning. Suchmodels may perform classification (e.g., binary or multiclassclassification), regression, clustering, dimensionality reduction,and/or such tasks. Examples of such models include, without limitation,artificial neural networks (ANN) (such as a recurrent neural networks(RNN) and convolutional neural network (CNN)), decision tree models(such as classification and regression trees (CART)), ensemble learningmodels (such as boosting, bootstrapped aggregation, gradient boostingmachines, and random forests), Bayesian network models (e.g., naiveBayes), principal component analysis (PCA), support vector machines(SVM), clustering models (such as K-nearest-neighbor, K-means,expectation maximization, hierarchical clustering, etc.), lineardiscriminant analysis models.

Any of the eye-tracking data generated by system 100 may be stored andhandled in a secure fashion (i.e., with respect to confidentiality,integrity, and availability). For example, a variety of symmetricaland/or asymmetrical encryption schemes and standards may be employed tosecurely handle the eye-tracking data at rest (e.g., in system 100) andin motion (e.g., when being transferred between the various modulesillustrated above). Without limiting the foregoing, such encryptionstandards and key-exchange protocols might include Triple DataEncryption Standard (3DES), Advanced Encryption Standard (AES) (such asAES-128, 192, or 256), Rivest-Shamir-Adelman (RSA), Twofish, RC4, RC5,RC6, Transport Layer Security (TLS), Diffie-Hellman key exchange, andSecure Sockets Layer (SSL). In addition, various hashing functions maybe used to address integrity concerns associated with the eye-trackingdata.

In addition, those skilled in the art will appreciate that embodimentsof the present disclosure may be practiced in conjunction with anynumber of systems, and that the systems described herein are merelyexemplary embodiments of the present disclosure. Further, the connectinglines shown in the various figures contained herein are intended torepresent example functional relationships and/or physical couplingsbetween the various elements. It should be noted that many alternativeor additional functional relationships or physical connections may bepresent in an embodiment of the present disclosure.

As used herein, the terms “module” or “controller” refer to anyhardware, software, firmware, electronic control component, processinglogic, and/or processor device, individually or in any combination,including without limitation: application specific integrated circuits(ASICs), field-programmable gate-arrays (FPGAs), dedicated neuralnetwork devices (e.g., Google Tensor Processing Units), electroniccircuits, processors (shared, dedicated, or group) configured to executeone or more software or firmware programs, a combinational logiccircuit, and/or other suitable components that provide the describedfunctionality.

As used herein, the word “exemplary” means “serving as an example,instance, or illustration.” Any implementation described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other implementations, nor is it intended to beconstrued as a model that must be literally duplicated.

While the foregoing detailed description will provide those skilled inthe art with a convenient road map for implementing various embodimentsof the invention, it should be appreciated that the particularembodiments described above are only examples, and are not intended tolimit the scope, applicability, or configuration of the invention in anyway. To the contrary, various changes may be made in the function andarrangement of elements described without departing from the scope ofthe invention.

1. A compact eye-tracking camera assembly for a computing device havinga display, the eye-tracking camera assembly including: an infrared imagesensor; a lens component; and a mirror component; wherein: the infraredimage sensor, lens component, and mirror component are substantiallycollinear along a principal axis, and are incorporated into the display,which is remote from and not fixed in relation to the face of a user;and the mirror component is oriented at a compound angle configured toredirect incident IR light along the principal axis, the compound anglecharacterized by a first angle of approximately 45 degrees relative to afirst axis, and a second angle of approximately 15 to 22 degreesrelative to a second axis that is orthogonal to the first axis, thefirst and second axes being substantially parallel to a plane defined bythe display.
 2. (canceled)
 3. The compact eye-tracking camera assemblyof claim 1, further including a front panel that is substantiallytransparent to IR light and at least partially opaque to visible light.4. The compact eye-tracking camera assembly of claim 1, furtherincluding a lens barrel configured to support the lens component.
 5. Thecompact eye-tracking camera assembly of claim 1, wherein the lenscomponent is supported by a structure having a first dimension that isless than or equal to the diameter of the lens component.
 6. The compacteye-tracking camera assembly of claim 1, wherein the total track lengthof the image sensor relative to the lens component is approximately 50%greater than the characteristic width of the lens component.
 7. Thecompact eye-tracking camera assembly of claim 6, wherein the lenscomponent has a diameter of approximately 10 mm, and the total tracklength is approximately 15 mm.
 8. A tablet computing device comprising:an enclosure; a display coupled to enclosure; and an eye-tracking cameraassembly incorporated into the display, which is remote from and notfixed in relation to the face of a user, the eye-tracking cameraassembly including: an infrared image sensor; a lens component; and amirror component; wherein the infrared image sensor, lens component, andmirror component are substantially collinear along a principal axis; andthe mirror component is oriented at a compound angle configured toredirect incident IR light along the principal axis, the compound anglecharacterized by a first angle of approximately 45 degrees relative to afirst axis and a second angle of approximately 15 to 22 degrees relativeto a second axis that is orthogonal to the first axis, the first andsecond axes being substantially parallel to a plane defined by thedisplay.
 9. (canceled)
 10. The compact eye-tracking camera assembly ofclaim 8, further including a front panel that is substantiallytransparent to IR light and at least partially opaque to visible light.11. The compact eye-tracking camera assembly of claim 8, furtherincluding a lens barrel configured to support the lens component. 12.The compact eye-tracking camera assembly of claim 8, wherein the lenscomponent is supported by a structure having a first dimension that isless than or equal to the diameter of the lens component.
 13. Thecompact eye-tracking camera assembly of claim 8, wherein the total tracklength of the image sensor relative to the lens component isapproximately 50% greater than the characteristic width of the lenscomponent.
 14. The compact eye-tracking camera assembly of claim 13,wherein the lens component has a diameter of approximately 10 mm, andthe total track length is approximately 15 mm.
 15. A method ofmanufacturing a tablet computing device, the method comprising:providing an enclosure; providing a display; forming an eye-trackingcamera assembly including an infrared image sensor, a lens component,and a mirror component, all of which are substantially collinear along aprincipal axis and are incorporated into the display, which is remotefrom and not fixed in relation to the face of a user; interconnectingthe enclosure, display, and eye-tracking camera assembly such that suchthat the mirror component is oriented at a compound angle configured toredirect incident IR light along the principal axis, the compound anglecharacterized by a first angle of approximately 45 degrees relative to afirst axis substantially parallel to a plane defined by the display, anda second angle of approximately 15-22 degrees relative to a second axisthat is orthogonal to the first axis, wherein the second axis issubstantially parallel to the plane defined by the display. 16.(canceled)
 17. The method of claim 15, further including attaching, overthe eye-tracking camera assembly, a front panel that is substantiallytransparent to IR light and at least partially opaque to visible light.18. The method of claim 15, further including supporting the lenscomponent with a lens barrel.
 19. The method of claim 15, wherein thetotal track length of the image sensor relative to the lens component isapproximately 50% greater than the characteristic width of the lenscomponent.
 20. The method of claim 19, wherein the lens component has adiameter of approximately 10 mm, and the total track length isapproximately 15 mm.